{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Multiclass classification with Amazon SageMaker XGBoost algorithm\n",
    "_**Single machine and distributed training for multiclass classification with Amazon SageMaker XGBoost algorithm**_\n",
    "\n",
    "---\n",
    "\n",
    "---\n",
    "## Contents\n",
    "\n",
    "1. [Introduction](#Introduction)\n",
    "2. [Prerequisites and Preprocessing](#Prequisites-and-Preprocessing)\n",
    "  1. [Permissions and environment variables](#Permissions-and-environment-variables)\n",
    "  2. [Data ingestion](#Data-ingestion)\n",
    "  3. [Data conversion](#Data-conversion)\n",
    "3. [Training the XGBoost model](#Training-the-XGBoost-model)\n",
    "  1. [Training on a single instance](#Training-on-a-single-instance)\n",
    "  2. [Training on multiple instances](#Training-on-multiple-instances)\n",
    "4. [Set up hosting for the model](#Set-up-hosting-for-the-model)\n",
    "  1. [Import model into hosting](#Import-model-into-hosting)\n",
    "  2. [Create endpoint configuration](#Create-endpoint-configuration)\n",
    "  3. [Create endpoint](#Create-endpoint)\n",
    "5. [Validate the model for use](#Validate-the-model-for-use)\n",
    "\n",
    "---\n",
    "## Introduction\n",
    "\n",
    "\n",
    "This notebook demonstrates the use of Amazon SageMaker’s implementation of the XGBoost algorithm to train and host a multiclass classification model. The MNIST dataset is used for training. It has a training set of 60,000 examples and a test set of 10,000 examples. To illustrate the use of libsvm training data format, we download the dataset and convert it to the libsvm format before training.\n",
    "\n",
    "To get started, we need to set up the environment with a few prerequisites for permissions and configurations.\n",
    "\n",
    "---\n",
    "## Prequisites and Preprocessing\n",
    "\n",
    "### Permissions and environment variables\n",
    "\n",
    "Here we set up the linkage and authentication to AWS services.\n",
    "\n",
    "1. The roles used to give learning and hosting access to your data. See the documentation for how to specify these.\n",
    "2. The S3 bucket that you want to use for training and model data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "isConfigCell": true
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "import os\n",
    "import boto3\n",
    "import re\n",
    "import copy\n",
    "import time\n",
    "from time import gmtime, strftime\n",
    "from sagemaker import get_execution_role\n",
    "\n",
    "role = get_execution_role()\n",
    "\n",
    "region = boto3.Session().region_name\n",
    "\n",
    "bucket='<bucket-name>' # put your s3 bucket name here, and create s3 bucket\n",
    "prefix = 'sagemaker/DEMO-xgboost-multiclass-classification'\n",
    "# customize to your bucket where you have stored the data\n",
    "bucket_path = 'https://s3-{}.amazonaws.com/{}'.format(region,bucket)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data ingestion\n",
    "\n",
    "Next, we read the dataset from the existing repository into memory, for preprocessing prior to training. This processing could be done *in situ* by Amazon Athena, Apache Spark in Amazon EMR, Amazon Redshift, etc., assuming the dataset is present in the appropriate location. Then, the next step would be to transfer the data to S3 for use in training. For small datasets, such as this one, reading into memory isn't onerous, though it would be for larger datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "import pickle, gzip, numpy, urllib.request, json\n",
    "\n",
    "# Load the dataset\n",
    "urllib.request.urlretrieve(\"http://deeplearning.net/data/mnist/mnist.pkl.gz\", \"mnist.pkl.gz\")\n",
    "f = gzip.open('mnist.pkl.gz', 'rb')\n",
    "train_set, valid_set, test_set = pickle.load(f, encoding='latin1')\n",
    "f.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data conversion\n",
    "\n",
    "Since algorithms have particular input and output requirements, converting the dataset is also part of the process that a data scientist goes through prior to initiating training. In this particular case, the data is converted from pickle-ized numpy array to the libsvm format before being uploaded to S3. The hosted implementation of xgboost consumes the libsvm converted data from S3 for training. The following provides functions for data conversions and file upload to S3 and download from S3. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "import struct\n",
    "import io\n",
    "import boto3\n",
    "\n",
    " \n",
    "def to_libsvm(f, labels, values):\n",
    "     f.write(bytes('\\n'.join(\n",
    "         ['{} {}'.format(label, ' '.join(['{}:{}'.format(i + 1, el) for i, el in enumerate(vec)])) for label, vec in\n",
    "          zip(labels, values)]), 'utf-8'))\n",
    "     return f\n",
    "\n",
    "\n",
    "def write_to_s3(fobj, bucket, key):\n",
    "    return boto3.Session(region_name=region).resource('s3').Bucket(bucket).Object(key).upload_fileobj(fobj)\n",
    "\n",
    "def get_dataset():\n",
    "  import pickle\n",
    "  import gzip\n",
    "  with gzip.open('mnist.pkl.gz', 'rb') as f:\n",
    "      u = pickle._Unpickler(f)\n",
    "      u.encoding = 'latin1'\n",
    "      return u.load()\n",
    "\n",
    "def upload_to_s3(partition_name, partition):\n",
    "    labels = [t.tolist() for t in partition[1]]\n",
    "    vectors = [t.tolist() for t in partition[0]]\n",
    "    num_partition = 5                                 # partition file into 5 parts\n",
    "    partition_bound = int(len(labels)/num_partition)\n",
    "    for i in range(num_partition):\n",
    "        f = io.BytesIO()\n",
    "        to_libsvm(f, labels[i*partition_bound:(i+1)*partition_bound], vectors[i*partition_bound:(i+1)*partition_bound])\n",
    "        f.seek(0)\n",
    "        key = \"{}/{}/examples{}\".format(prefix,partition_name,str(i))\n",
    "        url = 's3n://{}/{}'.format(bucket, key)\n",
    "        print('Writing to {}'.format(url))\n",
    "        write_to_s3(f, bucket, key)\n",
    "        print('Done writing to {}'.format(url))\n",
    "\n",
    "def download_from_s3(partition_name, number, filename):\n",
    "    key = \"{}/{}/examples{}\".format(prefix,partition_name, number)\n",
    "    url = 's3n://{}/{}'.format(bucket, key)\n",
    "    print('Reading from {}'.format(url))\n",
    "    s3 = boto3.resource('s3', region_name = region)\n",
    "    s3.Bucket(bucket).download_file(key, filename)\n",
    "    try:\n",
    "        s3.Bucket(bucket).download_file(key, 'mnist.local.test')\n",
    "    except botocore.exceptions.ClientError as e:\n",
    "        if e.response['Error']['Code'] == \"404\":\n",
    "            print('The object does not exist at {}.'.format(url))\n",
    "        else:\n",
    "            raise        \n",
    "        \n",
    "def convert_data():\n",
    "    train_set, valid_set, test_set = get_dataset()\n",
    "    partitions = [('train', train_set), ('validation', valid_set), ('test', test_set)]\n",
    "    for partition_name, partition in partitions:\n",
    "        print('{}: {} {}'.format(partition_name, partition[0].shape, partition[1].shape))\n",
    "        upload_to_s3(partition_name, partition)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "convert_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training the XGBoost model\n",
    "\n",
    "Now that we have our data in S3, we can begin training. We'll use Amazon SageMaker XGboost algorithm, and will actually fit two models in order to demonstrate the single machine and distributed training on SageMaker. In the first job, we'll use a single machine to train. In the second job, we'll use two machines and use the ShardedByS3Key mode for the train channel. Since we have 5 part file, one machine will train on three and the other on two part files. Note that the number of instances should not exceed the number of part files. \n",
    "\n",
    "First let's setup a list of training parameters which are common across the two jobs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sagemaker.amazon.amazon_estimator import get_image_uri\n",
    "container = get_image_uri(region, 'xgboost')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Ensure that the train and validation data folders generated above are reflected in the \"InputDataConfig\" parameter below.\n",
    "common_training_params = \\\n",
    "{\n",
    "    \"AlgorithmSpecification\": {\n",
    "        \"TrainingImage\": container,\n",
    "        \"TrainingInputMode\": \"File\"\n",
    "    },\n",
    "    \"RoleArn\": role,\n",
    "    \"OutputDataConfig\": {\n",
    "        \"S3OutputPath\": bucket_path + \"/\"+ prefix + \"/xgboost\"\n",
    "    },\n",
    "    \"ResourceConfig\": {\n",
    "        \"InstanceCount\": 1,   \n",
    "        \"InstanceType\": \"ml.m4.10xlarge\",\n",
    "        \"VolumeSizeInGB\": 5\n",
    "    },\n",
    "    \"HyperParameters\": {\n",
    "        \"max_depth\":\"5\",\n",
    "        \"eta\":\"0.2\",\n",
    "        \"gamma\":\"4\",\n",
    "        \"min_child_weight\":\"6\",\n",
    "        \"silent\":\"0\",\n",
    "        \"objective\": \"multi:softmax\",\n",
    "        \"num_class\": \"10\",\n",
    "        \"num_round\": \"10\"\n",
    "    },\n",
    "    \"StoppingCondition\": {\n",
    "        \"MaxRuntimeInSeconds\": 86400\n",
    "    },\n",
    "    \"InputDataConfig\": [\n",
    "        {\n",
    "            \"ChannelName\": \"train\",\n",
    "            \"DataSource\": {\n",
    "                \"S3DataSource\": {\n",
    "                    \"S3DataType\": \"S3Prefix\",\n",
    "                    \"S3Uri\": bucket_path + \"/\"+ prefix+ '/train/',\n",
    "                    \"S3DataDistributionType\": \"FullyReplicated\" \n",
    "                }\n",
    "            },\n",
    "            \"ContentType\": \"libsvm\",\n",
    "            \"CompressionType\": \"None\"\n",
    "        },\n",
    "        {\n",
    "            \"ChannelName\": \"validation\",\n",
    "            \"DataSource\": {\n",
    "                \"S3DataSource\": {\n",
    "                    \"S3DataType\": \"S3Prefix\",\n",
    "                    \"S3Uri\": bucket_path + \"/\"+ prefix+ '/validation/',\n",
    "                    \"S3DataDistributionType\": \"FullyReplicated\"\n",
    "                }\n",
    "            },\n",
    "            \"ContentType\": \"libsvm\",\n",
    "            \"CompressionType\": \"None\"\n",
    "        }\n",
    "    ]\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we'll create two separate jobs, updating the parameters that are unique to each.\n",
    "\n",
    "### Training on a single instance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#single machine job params\n",
    "single_machine_job_name = 'DEMO-xgboost-classification' + strftime(\"%Y-%m-%d-%H-%M-%S\", gmtime())\n",
    "print(\"Job name is:\", single_machine_job_name)\n",
    "\n",
    "single_machine_job_params = copy.deepcopy(common_training_params)\n",
    "single_machine_job_params['TrainingJobName'] = single_machine_job_name\n",
    "single_machine_job_params['OutputDataConfig']['S3OutputPath'] = bucket_path + \"/\"+ prefix + \"/xgboost-single\"\n",
    "single_machine_job_params['ResourceConfig']['InstanceCount'] = 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training on multiple instances\n",
    "\n",
    "You can also run the training job distributed over multiple instances. For larger datasets with multiple partitions, this can significantly boost the training speed. Here we'll still use the small/toy MNIST dataset to demo this feature."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#distributed job params\n",
    "distributed_job_name = 'DEMO-xgboost-distrib-classification' + strftime(\"%Y-%m-%d-%H-%M-%S\", gmtime())\n",
    "print(\"Job name is:\", distributed_job_name)\n",
    "\n",
    "distributed_job_params = copy.deepcopy(common_training_params)\n",
    "distributed_job_params['TrainingJobName'] = distributed_job_name\n",
    "distributed_job_params['OutputDataConfig']['S3OutputPath'] = bucket_path + \"/\"+ prefix + \"/xgboost-distributed\"\n",
    "#number of instances used for training\n",
    "distributed_job_params['ResourceConfig']['InstanceCount'] = 2 # no more than 5 if there are total 5 partition files generated above\n",
    "\n",
    "# data distribution type for train channel\n",
    "distributed_job_params['InputDataConfig'][0]['DataSource']['S3DataSource']['S3DataDistributionType'] = 'ShardedByS3Key'\n",
    "# data distribution type for validation channel\n",
    "distributed_job_params['InputDataConfig'][1]['DataSource']['S3DataSource']['S3DataDistributionType'] = 'ShardedByS3Key'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's submit these jobs, taking note that the first will be submitted to run in the background so that we can immediately run the second in parallel."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "sm = boto3.Session(region_name=region).client('sagemaker')\n",
    "\n",
    "sm.create_training_job(**single_machine_job_params)\n",
    "sm.create_training_job(**distributed_job_params)\n",
    "\n",
    "status = sm.describe_training_job(TrainingJobName=distributed_job_name)['TrainingJobStatus']\n",
    "print(status)\n",
    "sm.get_waiter('training_job_completed_or_stopped').wait(TrainingJobName=distributed_job_name)\n",
    "status = sm.describe_training_job(TrainingJobName=distributed_job_name)['TrainingJobStatus']\n",
    "print(\"Training job ended with status: \" + status)\n",
    "if status == 'Failed':\n",
    "    message = sm.describe_training_job(TrainingJobName=distributed_job_name)['FailureReason']\n",
    "    print('Training failed with the following error: {}'.format(message))\n",
    "    raise Exception('Training job failed')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's confirm both jobs have finished."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "print('Single Machine:', sm.describe_training_job(TrainingJobName=single_machine_job_name)['TrainingJobStatus'])\n",
    "print('Distributed:', sm.describe_training_job(TrainingJobName=distributed_job_name)['TrainingJobStatus'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Set up hosting for the model\n",
    "In order to set up hosting, we have to import the model from training to hosting. The step below demonstrated hosting the model generated from the distributed training job. Same steps can be followed to host the model obtained from the single machine job. \n",
    "\n",
    "### Import model into hosting\n",
    "Next, you register the model with hosting. This allows you the flexibility of importing models trained elsewhere."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "import boto3\n",
    "from time import gmtime, strftime\n",
    "\n",
    "model_name=distributed_job_name + '-mod'\n",
    "print(model_name)\n",
    "\n",
    "info = sm.describe_training_job(TrainingJobName=distributed_job_name)\n",
    "model_data = info['ModelArtifacts']['S3ModelArtifacts']\n",
    "print(model_data)\n",
    "\n",
    "primary_container = {\n",
    "    'Image': container,\n",
    "    'ModelDataUrl': model_data\n",
    "}\n",
    "\n",
    "create_model_response = sm.create_model(\n",
    "    ModelName = model_name,\n",
    "    ExecutionRoleArn = role,\n",
    "    PrimaryContainer = primary_container)\n",
    "\n",
    "print(create_model_response['ModelArn'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create endpoint configuration\n",
    "SageMaker supports configuring REST endpoints in hosting with multiple models, e.g. for A/B testing purposes. In order to support this, customers create an endpoint configuration, that describes the distribution of traffic across the models, whether split, shadowed, or sampled in some way. In addition, the endpoint configuration describes the instance type required for model deployment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from time import gmtime, strftime\n",
    "\n",
    "endpoint_config_name = 'DEMO-XGBoostEndpointConfig-' + strftime(\"%Y-%m-%d-%H-%M-%S\", gmtime())\n",
    "print(endpoint_config_name)\n",
    "create_endpoint_config_response = sm.create_endpoint_config(\n",
    "    EndpointConfigName = endpoint_config_name,\n",
    "    ProductionVariants=[{\n",
    "        'InstanceType':'ml.m4.xlarge',\n",
    "        'InitialVariantWeight':1,\n",
    "        'InitialInstanceCount':1,\n",
    "        'ModelName':model_name,\n",
    "        'VariantName':'AllTraffic'}])\n",
    "\n",
    "print(\"Endpoint Config Arn: \" + create_endpoint_config_response['EndpointConfigArn'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create endpoint\n",
    "Lastly, the customer creates the endpoint that serves up the model, through specifying the name and configuration defined above. The end result is an endpoint that can be validated and incorporated into production applications. This takes 9-11 minutes to complete."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "import time\n",
    "\n",
    "endpoint_name = 'DEMO-XGBoostEndpoint-' + strftime(\"%Y-%m-%d-%H-%M-%S\", gmtime())\n",
    "print(endpoint_name)\n",
    "create_endpoint_response = sm.create_endpoint(\n",
    "    EndpointName=endpoint_name,\n",
    "    EndpointConfigName=endpoint_config_name)\n",
    "print(create_endpoint_response['EndpointArn'])\n",
    "\n",
    "resp = sm.describe_endpoint(EndpointName=endpoint_name)\n",
    "status = resp['EndpointStatus']\n",
    "print(\"Status: \" + status)\n",
    "\n",
    "while status=='Creating':\n",
    "    time.sleep(60)\n",
    "    resp = sm.describe_endpoint(EndpointName=endpoint_name)\n",
    "    status = resp['EndpointStatus']\n",
    "    print(\"Status: \" + status)\n",
    "\n",
    "print(\"Arn: \" + resp['EndpointArn'])\n",
    "print(\"Status: \" + status)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Validate the model for use\n",
    "Finally, the customer can now validate the model for use. They can obtain the endpoint from the client library using the result from previous operations, and generate classifications from the trained model using that endpoint.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "runtime_client = boto3.client('runtime.sagemaker', region_name=region)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to evaluate the model, we'll use the test dataset previously generated. Let us first download the data from S3 to the local host."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "download_from_s3('test', 0, 'mnist.local.test') # reading the first part file within test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Start with a single prediction. Lets use the first record from the test file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "!head -1 mnist.local.test > mnist.single.test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "import json\n",
    "\n",
    "file_name = 'mnist.single.test' #customize to your test file 'mnist.single.test' if use the data above\n",
    "\n",
    "with open(file_name, 'r') as f:\n",
    "    payload = f.read()\n",
    "\n",
    "response = runtime_client.invoke_endpoint(EndpointName=endpoint_name, \n",
    "                                   ContentType='text/x-libsvm', \n",
    "                                   Body=payload)\n",
    "result = response['Body'].read().decode('ascii')\n",
    "print('Predicted label is {}.'.format(result))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "OK, a single prediction works.\n",
    "Let's do a whole batch and see how good is the predictions accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "def do_predict(data, endpoint_name, content_type):\n",
    "    payload = '\\n'.join(data)\n",
    "    response = runtime_client.invoke_endpoint(EndpointName=endpoint_name, \n",
    "                                   ContentType=content_type, \n",
    "                                   Body=payload)\n",
    "    result = response['Body'].read().decode('ascii')\n",
    "    preds = [float(num) for num in result.split(',')]\n",
    "    return preds\n",
    "\n",
    "def batch_predict(data, batch_size, endpoint_name, content_type):\n",
    "    items = len(data)\n",
    "    arrs = []\n",
    "    for offset in range(0, items, batch_size):\n",
    "        arrs.extend(do_predict(data[offset:min(offset+batch_size, items)], endpoint_name, content_type))\n",
    "        sys.stdout.write('.')\n",
    "    return(arrs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following function helps us calculate the error rate on the batch dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "import json\n",
    "\n",
    "file_name = 'mnist.local.test'\n",
    "with open(file_name, 'r') as f:\n",
    "    payload = f.read().strip()\n",
    "\n",
    "labels = [float(line.split(' ')[0]) for line in payload.split('\\n')]\n",
    "test_data = payload.split('\\n')\n",
    "preds = batch_predict(test_data, 100, endpoint_name, 'text/x-libsvm')\n",
    "\n",
    "print ('\\nerror rate=%f' % ( sum(1 for i in range(len(preds)) if preds[i]!=labels[i]) /float(len(preds))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are a few predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "preds[0:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and the corresponding labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "labels[0:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following function helps us create the confusion matrix on the labeled batch test dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy\n",
    "def error_rate(predictions, labels):\n",
    "    \"\"\"Return the error rate and confusions.\"\"\"\n",
    "    correct = numpy.sum(predictions == labels)\n",
    "    total = predictions.shape[0]\n",
    "\n",
    "    error = 100.0 - (100 * float(correct) / float(total))\n",
    "\n",
    "    confusions = numpy.zeros([10, 10], numpy.int32)\n",
    "    bundled = zip(predictions, labels)\n",
    "    for predicted, actual in bundled:\n",
    "        confusions[int(predicted), int(actual)] += 1\n",
    "    \n",
    "    return error, confusions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following helps us visualize the erros that the XGBoost classifier is making. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline  \n",
    "\n",
    "NUM_LABELS = 10  # change it according to num_class in your dataset\n",
    "test_error, confusions = error_rate(numpy.asarray(preds), numpy.asarray(labels))\n",
    "print('Test error: %.1f%%' % test_error)\n",
    "\n",
    "plt.xlabel('Actual')\n",
    "plt.ylabel('Predicted')\n",
    "plt.grid(False)\n",
    "plt.xticks(numpy.arange(NUM_LABELS))\n",
    "plt.yticks(numpy.arange(NUM_LABELS))\n",
    "plt.imshow(confusions, cmap=plt.cm.jet, interpolation='nearest');\n",
    "\n",
    "for i, cas in enumerate(confusions):\n",
    "    for j, count in enumerate(cas):\n",
    "        if count > 0:\n",
    "            xoff = .07 * len(str(count))\n",
    "            plt.text(j-xoff, i+.2, int(count), fontsize=9, color='white')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Delete Endpoint\n",
    "Once you are done using the endpoint, you can use the following to delete it. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sm.delete_endpoint(EndpointName=endpoint_name)"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  },
  "notice": "Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved. Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance with the License. A copy of the License is located at http://aws.amazon.com/apache2.0/ or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License."
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
